In recent years, mobile devices such as cell-phones, tablets and laptops have become ubiquitous. Most of these devices include one or two compact cameras—a main rear-facing camera (i.e. a camera on the back side of the device, facing away from the user and often used for casual photography) and a secondary front-facing camera (i.e. a camera located on the front side of the device and often used for video conferencing).
Although relatively compact in nature, the design of most of these cameras is very similar to the traditional structure of a digital still camera, i.e. they comprise an optical component (or a train of several optical elements and a main aperture) placed on top of an image sensor. The optical component (also referred to as “optics”) refracts the incoming light rays and bends them to create an image of a scene on the sensor. The dimensions of these cameras are largely determined by the size of the sensor and by the height of the optics. These are usually tied together through the focal length (“f”) of the lens and its field of view (FOV)—a lens that has to image a certain FOV on a sensor of a certain size has a specific focal length. Keeping the FOV constant, the larger the sensor dimensions (e.g. in an X-Y plane), the larger the focal length and the optics height.
As the dimensions of mobile devices shrink, the compact camera dimensions become more and more a key factor that limits the device thickness. Several approaches have been proposed to reduce the compact camera thickness in order to alleviate this constraint. Recently, multi-aperture systems have been proposed for this purpose. In such systems, instead of having one aperture with one train of optical elements, the camera is divided into several apertures, each with dedicated optical elements, all apertures sharing a similar field of view. Hereinafter, each such aperture, together with the optics and the sensor area on which the image is formed, is defined as a “sub-camera”. Typically, in multi-aperture camera designs, each sub-camera creates a smaller image on the image sensor compared with the image created by a reference single-aperture camera. Therefore, the height of each sub-camera can be smaller than the height of a single-aperture camera, reducing the total height of the camera could be reduced and allowing for slimmer designs of mobile devices.
FIG. 1A and FIG. 1B show a schematic design of a traditional camera and of a dual-aperture camera with two sub-cameras, respectively. A traditional camera 100′ in FIG. 1A includes an image sensor 102 placed on a substrate 104 and a lens 106. A “camera height” is defined as the height of the camera module, from substrate 104 to the top of lens 106. A dual-aperture camera 100″ in FIG. 1B includes two sub-cameras, a sub-camera 1 with an image sensor 112a and a lens 116a with an optical axis 118a, and a sub-camera 2 with, an image sensor 112b and a lens 116b with an optical axis 118b. The two sensors are placed on, respectively, substrates 114a and 114b. For comparison's sake, it is assumed that the reference single-aperture camera and the dual-aperture camera have the same field of view (FOV) and the sensors have the same pixel size. However, image sensor 102 has a higher resolution (number of pixels) compared with image sensor 112a or image sensor 112b, and is therefore larger in size. The potential advantage in camera height of the dual-aperture camera (i.e. the thickness from substrate 114a to the top of lens 116a and from substrate 114b to the top of lens 116b) may be appreciated.
There are several significant challenges involved in multi-aperture camera designs. First and foremost, the sensor area of each sub-camera is smaller compared with that of a single-aperture camera. If the pixel size in each sub-camera sensor is kept the same as that in the single-aperture camera sensor, the resolution of an image captured by each sub-camera is smaller than that captured by the single-aperture camera. If the resolution of the output image is to be kept the same, the images from the different sub-cameras need to be combined into a higher-resolution image. This is usually done in the digital domain, by a dedicated algorithm. Several methods have been proposed for combining lower-resolution images to produce a higher-resolution image. Some algorithms in such methods require a registration step between the set of low-resolution images, to account for parallax (which is present in a multi-aperture camera system due to the shift in point-of-view between sub-cameras). One such algorithm is described in co-assigned PCT patent application PCT/IB2014/062180 titled “Dual aperture zoom digital camera”, which is incorporated herein by reference in its entirety.
Another challenge relates to the requirement that the camera provides an in-focus image for a wide range of object distances (usually from several centimeters to infinity in compact camera modules). To fulfill this requirement, a single-aperture camera may include an Auto-Focus (AF) mechanism that controls the focus position of the optics, by moving the optical element along the optical axis, thus changing its height above the sensor. In multi-aperture cameras, in order to support an in-focus image for a wide range of object distances, a straightforward approach would be to provide a dedicated AF mechanism in each sub-camera. This approach has several drawbacks including increased size and cost of the camera, higher operating power and more complicated control, as the AF mechanisms of each sub-camera needs to be synchronized, to ensure all of the sub-cameras are focused to the same position.
Another complication that may arise when using an AF mechanism in a multi-aperture camera is connected with the algorithm that combines the lower resolution sub-camera images to produce a higher resolution image. Since an AF mechanism moves the optical element along the optical axis above the sensor, it scales the image that is formed on the sensor to some extent. Slight differences between the focusing positions of different AF mechanisms in each sub-camera may result in different scales applied to the lower resolution sub-camera images. Such differences in scale may degrade the performance of the image registration step in the algorithm. Correcting for the different scale is not trivial, due to the dynamic nature of the scale—the scale applied on the image depends on the focus position of the optics, which in turn changes with object distance. This means that the scale cannot be trivially corrected by calibrating the multi-aperture camera and applying a fixed correction, but rather, the correct scale has to be estimated at each image. Estimating the correct scale to apply from the image is not trivial, in the presence of parallax (where different objects appear at different locations as a function from their distance from the camera) and in the presence of possible occlusions of objects in one aperture but not in the other. There is therefore a need for a method that can accurately estimate and correct differences in scaling on a per-image basis.
As an alternative to using AF, multi-aperture camera designs have been proposed with no AF mechanism at all. Such designs rely on the smaller focal length of each sub-camera to provide increased depth-of-focus (DOF) compared with a corresponding single-aperture camera that supports a larger sensor. Since a larger DOF means that a wider range of object distances is imaged in-focus onto the sensor, the AF mechanism could be removed. While this approach is advantageous in terms of cost, size and system complexity, the larger DOF that results from the shorter focal length of a multi-aperture camera is often insufficient to support an in-focus image for object distances ranging from a few centimeters to infinity. In these cases, settling for a multi-aperture camera with fixed-focus optics results in poor imaging performance at close object distances.
Between using multiple AF mechanisms and using only fixed-focus optics, there is a need for a multi-aperture camera system that combines the benefits of an AF mechanism without adding additional complexity and cost to the camera system.